Parameter Settings Optimization in Map Reduce Big Data processing using the MOPSO Algorithm
نویسندگان
چکیده
Big data is a commodity that highly valued in the entire globe. It not just regarded as but world of experts, we can derive intelligence from it. Because its characteristics which are Variety, Value, Volume, Velocity, and growing need how it be handled, Organizations facing difficulties ensuring optimal well affordable processing storage large datasets. One already existing models used for rapid together with big known Hadoop MapReduce. MapReduce large-scale parallel distributed computing environment, while running applications storing clusters hardware Furthermore, framework needs to tune more than 190 configuration parameters mostly done manually. Due complex interactions spaces between parameters, manual tuning effective. Even worse, these must tuned every time run. The main goal this research create an algorithm will improve efficiency by automatically optimizing parameter settings when jobs running. employs Multi-Objective Particle Swarm Optimization (MOPSO) technique, uses two objective functions look Pareto solution parameters. results experiments have shown has remarkably improved job performance comparison use default settings.
منابع مشابه
Big Data Processing with Hadoop Map-reduce
The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data ...
متن کاملFeature Selection in Structural Health Monitoring Big Data Using a Meta-Heuristic Optimization Algorithm
This paper focuses on the processing of structural health monitoring (SHM) big data. Extracted features of a structure are reduced using an optimization algorithm to find a minimal subset of salient features by removing noisy, irrelevant and redundant data. The PSO-Harmony algorithm is introduced for feature selection to enhance the capability of the proposed method for processing the measure...
متن کاملSTRUCTURAL OPTIMIZATION USING BIG BANG-BIG CRUNCH ALGORITHM: A REVIEW
The big bang-big crunch (BB-BC) algorithm is a popular metaheuristic optimization technique proposed based on one of the theories for the evolution of the universe. The algorithm utilizes a two-phase search mechanism: big-bang phase and big-crunch phase. In the big-bang phase the concept of energy dissipation is considered to produce disorder and randomness in the candidate population while in ...
متن کاملSignificant Big Data Interpretation using Map Reduce Paradigm
The development of ontologies involves continuous but relatively small modifications. Even after a number of changes, ontology and its previous versions usually share most of their axioms. For large and complex ontologies this may require a few minutes, or even a few hours. Cognitive on a Web scale becomes increasingly stimulating because of the large volume of data involved and the complexity ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: International journal of advances in scientific research and engineering
سال: 2021
ISSN: ['2454-8006']
DOI: https://doi.org/10.31695/ijasre.2021.33923